Input control device, input control method, and operation system

ABSTRACT

The present technology relates to an input control device, an input control method, and an operation system, which enable achievement of a highly flexible input operation without a burden on a user. A target point determination unit determines a target point to be a display reference in an operative field image in a display, on the basis of a user&#39;s line-of-sight position with respect to the operative field image displayed in the display and of a characteristic point detected in the operative field image. The present technology can be applied to an input device for an operation system.

TECHNICAL FIELD

The present technology relates to an input control device, an inputcontrol method, and an operation system, in particular, to an inputcontrol device, an input control method, and an operation system thatenable achievement of a highly flexible input operation.

BACKGROUND ART

In recent years, it has been proposed for operation systems that screenoperations are performed by zoom control of a camera or drive control ofa camera arm by a natural user interface (NUI) of a user (operator),such as lines of sight with respect to an operative field image.

Furthermore, conventionally, screen operations have been performed byzoom control of a camera or drive control of a camera arm by regarding,as a target point, a characteristic point of an operation instrument orthe like displayed in an operative field image acquired by the camera.

For example, Patent Document 1 discloses that, in an endoscopicoperation system, a marker is provided in the vicinity of a tip of atreatment device. Display contents of the marker are changeable by usingswitches provided at a hand-side of the treatment device, and anendoscope is operated on the basis of the change in the display contentsof the marker in an observation image from the endoscope.

A screen operation by a line-of-sight position allows a user to selectan arbitrary position in a screen as the target point without using ahand. However, the user has not been able to operate the screen whilecomprehensively observing an operative field, because the user isinevitably not able to observe positions other than a gaze point.

On the other hand, a screen operation regarding, as the target point,the characteristic point of an operation instrument or the like allowthe user to observe a position other than the target point. However,although an operation instrument needs to be moved for observation of awider operative field, the user has sometimes not been able to perform adesirable screen operation in a case where the movement of the operationinstrument is limited, for example, a case where an operation instrumentis holding an organ.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No. 9-28713

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Therefore, it is desirable that the target point used for a screenoperation can be switched depending on a situation. However, there hasbeen no method for achieving the switching by a simple input operationwithout a burden on a user.

The present technology has been developed to solve the problemsmentioned above, and an object of the present technology is to enableachievement of a highly flexible input operation without a burden on auser.

Solutions to Problems

An input control device according to the present technology includes atarget point determination unit that determines a target point to be adisplay reference in an operative field image in a display on the basisof a user's line-of-sight position with respect to the operative fieldimage displayed in the display and of a characteristic point detected inthe operative field image.

An input control method according to the present technology includes astep of determining, by an input control device, a target point to be adisplay reference in an operative field image in a display on the basisof a user's line-of-sight position with respect to the operative fieldimage displayed in the display and of a characteristic point detected inthe operative field image.

An operation system according to the present technology includes acamera that captures an operative field image, a display that displaysthe operative field image, and an input control device including atarget point determination unit that determines a target point to be adisplay reference in the operative field image in the display on thebasis of a user's line-of-sight position with respect to the operativefield image displayed in the display and of a characteristic pointdetected in the operative field image.

In the present technology, a target point to be a display reference inan operative field image in a display is determined on the basis of auser's line-of-sight position with respect to the operative field imagedisplayed in the display and of a characteristic point detected in theoperative field image.

Effects of the Invention

The present technology enables achievement of a highly flexible inputoperation without a burden on a user. Note that the effects describedherein are not necessarily limited, and any one of effects described inthe present disclosure may be applied.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of an operationsystem, to which the present technology is applied.

FIG. 2 is a top view illustrating a configuration example of appearanceof a foot switch.

FIG. 3 is a diagram illustrating examples of screen operations in theoperation system.

FIG. 4 is a view illustrating specific examples of screen operationsbased on a target point.

FIG. 5 is a block diagram illustrating a functional configurationexample of the operation system.

FIG. 6 is a flowchart describing a first example of operation controlprocessing.

FIG. 7 is a view illustrating an example of target point determination.

FIG. 8 is a view illustrating an example of target point determination.

FIG. 9 is a view describing an example of a feedback of a target pointcandidate to a user.

FIG. 10 is a view illustrating an example of a screen display in anevent of operation control stop.

FIG. 11 is a flowchart describing a second example of the operationcontrol processing.

FIG. 12 is a view illustrating an example of target point selection.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, a mode (hereinafter referred to as embodiment) for carryingout the present disclosure will be described. Note that the descriptionwill be made in the following order.

1. Outline of Operation System

2. Functional Configuration Example of Operation System

3. First Example of Operation Control Processing of Operation System

4. Second Example of Operation Control Processing of Operation System

<1. Outline of Operation System>

FIG. 1 is a block diagram illustrating a configuration example of anoperation system, to which the present technology is applied.

An operation system 10 includes an operative field camera 11, a cameraarm 12, a motion recognition camera 13, a display 14, a control device15, an operating table 16, a microphone 17, and a foot switch 18. Theoperation system 10 is arranged in an operating room or the like andenables treatment such as surgery with reference to an image captured bythe operative field camera 11.

The operative field camera 11 (an image capturing device for operation)is a modality device, such as a 3D camera, supported by the camera arm12. The operative field camera 11 captures an image of an operativefield and the like of a patient 21 laid on the operating table 16 andtransmits a 3D image acquired as a result of the image capturing to thecontrol device 15, as an operative field image. The camera arm 12supports the operative field camera 11 and controls a position and anangle of the operative field camera 11.

The motion recognition camera 13 is, for example, a 2D camera and isarranged on the display 14. The motion recognition camera 13 captures animage of an operator 22 wearing the microphone 17 on a head 22A. Themotion recognition camera 13 transmits a 2D image acquired as a resultof the image capturing to the control device 15, as an operator's image.

The display 14 is a 3D display having a relatively small screen and isarranged relatively near the operator 22 (in the example of FIG. 1,above the operating table 16 and near the operator 22). The display 14displays the operative field image and the like transmitted from thecontrol device 15.

The control device 15 sets a control mode of the operation system 10 toa manual operation mode or a hands-free mode. The manual operation modeis a mode to control the operation system 10 on the basis of input (forexample, force applied to the camera arm 12 or an operation of anunillustrated control button or the like provided at each unit) by ahand of the operator 22. The hands-free mode is a mode to control theoperation system 10, not by using the hand of the operator 22, but onthe basis of contactless input by sound, a line of sight, movement or adirection of the head 22A, a gesture, and the like, or input by a leg22B touching the foot switch 18.

An operation, mainly in a case where the control mode is in thehands-free mode, will be described hereinafter.

The control device 15 recognizes movement or a direction of the head 22Aby detecting a position of the head 22A in the operator's imagetransmitted from the motion recognition camera 13. Furthermore, thecontrol device 15 detects a direction of a line of sight of the operator22 from the operator's image and, on the basis of a direction of theline of sight, recognizes a position of a line of sight on a screen ofthe display 14.

Note that, in the operation system 10, the line of sight is detected byusing the operator's image captured by the motion recognition camera 13.However, the operator 22 may wear glasses equipped with a line-of-sightdetection device, and the line-of-sight detection device may detect theline of sight.

Furthermore, in the operation system 10, movement or a direction of thehead 22A is detected from the operator's image, because a distancebetween the motion recognition camera 13 and the operator 22 is short.However, the operator 22 may wear a marker, and the movement or thedirection of the head 22A may be detected from a position of the markerin the operator's image.

The control device 15 receives sound transmitted from the microphone 17and recognizes the sound. The control device 15 receives an operationsignal which is transmitted from the foot switch 18 and shows anoperation on the foot switch 18, and, on the basis of the operationsignal, recognizes content of the operation on the foot switch 18.

Moreover, in a case where the control mode is in the hands-free mode,the control device 15 regards, as input information, movement and adirection of the head 22A, a gesture of the operator 22, information ofthe line-of-sight position showing a position of a line of sight on thescreen of the display 14, a sound recognition result, volume, andoperation information showing content of the operation on the footswitch 18. The control device 15 recognizes a command from the operator22 and a state of the operator 22, on the basis of the inputinformation.

The control device 15 permits the command from the operator 22 accordingto a state of the operator 22. According to the permitted command, thecontrol device 15 controls image capturing with the operative fieldcamera 11, controls a drive of the camera arm 12, controls a display ofthe display 14, or changes the control mode.

The microphone 17 is worn on the head 22A of the operator 22. Themicrophone 17 acquires a surrounding sound including voice of theoperator 22 and transmits the sound to the control device 15.

The foot switch 18 is arranged around the operator 22 and operated bythe touch by the leg 22B of the operator 22. The foot switch 18transmits, to the control device 15, the operation signal showing theoperation by the leg 22B of the operator 22.

In the operation system 10 configured as above, the operator 22 lays thepatient 21 on the operating table 16 and performs treatment such assurgery while checking the operative field image and the like displayedin the display 14.

Furthermore, the operator 22 performs contactless input or input by atouch by a foot, when changing the control mode, an image capturecondition of the operative field camera 11, a position and an angle ofthe operative field camera 11, a display of the display 14, or the like.Therefore, the operator 22 can perform input while holding anunillustrated operation instrument. Therefore, the operator 22 does notrequire sterilization treatment each time performing input.

Note that any method can be adopted for the detection of the line ofsight, for the detection of the movement or the direction of the head22A of the operator 22 and the gesture of the operator 22, and foracquisition of the sound. For example, the line-of-sight detectiondevice and the microphone 17 may not necessarily be a wearable device.

(Examples of Appearance Configuration and Screen Operation of FootSwitch)

FIG. 2 is a top view illustrating a configuration example of appearanceof the foot switch 18.

The foot switch 18 includes, as an operation input part, a lever 31 anda treading part 32.

The lever 31 is configured as a so-called joystick and receivesoperation input by the leg 22B. The lever 31 is used for, for example,positioning of the operative field camera 11 in the manual operationmode.

The treading part 32 is generally configured as a physical button andreceives the operation input by the leg 22B. The treading part 32 isused to control each part of the operation system 10. In the presentembodiment, an operation mode is set according to movement of theoperator 22 and a position of the treading part 32 stepped on by the leg22B. Then, according to the set operation mode, the operative fieldcamera 11 and the camera arm 12 are controlled, by which a screenoperation of the operative field image displayed in the display 14 isperformed.

For example, if an upper left part 32 a or an upper right part 32 b ofthe treading part 32 is stepped on, as illustrated in an upper part ofFIG. 3, the operative field image is zoomed in or out while apredetermined target point in the operative field image displayed in thedisplay 14 is moving to a center of the screen (operation mode: ZOOM).

Furthermore, if the head 22A of the operator 22 tilts to the left orright while a lower left part 32 c or a lower right part 32 d of thetreading part 32 is being stepped on, as illustrated in a middle part ofFIG. 3, an image capturing angle of the operative field image is changedwhile the predetermined target point in the operative field imagedisplayed in the display 14 is moving to the center of the screen(operation mode: PIVOT).

Moreover, if the lower left part 32 c or a lower right part 32 d of thetreading part 32 is stepped on, as illustrated in a lower part of FIG.3, the predetermined target point in the operative field image displayedin the display 14 moves to the center of the screen (operation mode:MOVE).

Note that any operation mode may be set as an operation mode, notlimited to each operation of ZOOM, PIVOT, and MOVE, described above.

(Specific Examples of Screen Operation Based on Target Point)

Here, with reference to FIG. 4, specific examples of a screen operationbased on the target point will be described.

In the operation system 10, a target point to be a display reference inthe operative field image is determined, and drive control of the cameraarm 12 is performed sequentially until the target point moves to thecenter of the screen.

For example, as illustrated in FIG. 4, in a case where a line-of-sightposition E1 of the user to the operative field image is regarded as thetarget point, the camera arm 12 performs a following motion so that theline-of-sight position E1 is positioned at the center of the screen. Atthis time, the camera arm 12 performs the following motion not by thescreen operation described with reference to FIG. 3. For example, in theZOOM operation, the camera arm 12 performs the following motion whilethe operative field image is being zoomed in, so that the line-of-sightposition E1 is positioned at the center of the image. Meanwhile in theMOVE operation, the camera arm 12 performs the following motion so that,simply, the line-of-sight position E1 is positioned at the center of thescreen.

<2. Functional Configuration Example of Operation System>

FIG. 5 is a block diagram illustrating a functional configurationexample of the operation system 10.

The operation system 10 illustrated in FIG. 5 includes the operativefield camera 11, the camera arm 12, the display 14, a user interface(I/F) 51, an input control device 52, and a control device 71.

The user I/F 51 includes various input devices in the operation system10. The user (operator 22) can input various kinds of information orinstructions into the operation system 10 via the user I/F 51. The inputinformation indicating content of the input into the user I/F 51 isprovided to the input control device 52.

The user I/F 51 includes the motion recognition camera 13, themicrophone 17, and the foot switch 18. The motion recognition camera 13provides the input control device 52 with the operator's image, which isthe captured image of the operator, as input information. Furthermore,the foot switch 18 provides the input control device 52 with operationinformation showing content of the operation by the operator 22, asinput information.

Moreover, the user I/F 51 may include a device, for example, aglasses-type wearable device or a head mounted display (HMD), which isworn by the user, and various kinds of input may be performed accordingto the gesture or the line of sight of the user detected by thesedevices.

The input control device 52, on the basis of the input information fromthe user I/F 51, generates control information that the control device71 will use to control the operative field camera 11 and the camera arm12, and then provides the control information to the control device 71.

The input control device 52 includes a target point determination unit61, a parameter generation unit 62, and an operation mode determinationunit 63.

The target point determination unit 61 determines, on the basis of theinput information provided from the user I/F 51, the target point to bethe display reference in the operative field image in the display 14.

Specifically, the target point determination unit 61 detects thedirection of the line of sight of the operator 22 from the operator'simage provided as the input information by the user I/F 51, and, on thebasis of the direction, recognizes the line-of-sight position on thescreen of the display 14. Furthermore, the target point determinationunit 61 acquires information which is provided by the control device 71and shows a characteristic point detected in the operative field imagecaptured by the operative field camera 11. Then, the target pointdetermination unit 61 determines any one of the line-of-sight positionof the operator 22 or the characteristic point detected in the operativefield image, as the target point to be the display reference in thedisplay 14.

The parameter generation unit 62 generates a parameter to control theoperative field camera 11 and the camera arm 12, in order to display theoperative field image in the display 14 with reference to the targetpoint determined by the target point determination unit 61. Theparameter generated by the parameter generation unit 62 is provided tothe control device 71 as control information.

By detecting the position of the head 22A in the operator's imageprovided as the input information by the user I/F 51, the operation modedetermination unit 63 recognizes movement of the operator 22, such asmovement or a direction of the head 22A. The operation modedetermination unit 63 determines, on the basis of a recognition resultof the movement of the operator 22, an operation mode (ZOOM, PIVOT, orMOVE) being instructed by the operator 22 and provides the controldevice 71 with operation mode information showing the operation mode, ascontrol information.

On the basis of the control information provided by the input controldevice 52, the control device 71 controls operation of the operativefield camera 11 and the camera arm 12, as well as a display of theoperative field image in the display 14.

The control device 71 includes a camera control unit (CCU) 81 and an armcontrol unit 82.

The CCU 81 controls an operation of the operative field camera 11 andthe display 14.

Specifically, on the basis of the parameter from the parametergeneration unit 62 and the operation mode information from the operationmode determination unit 63, the CCU 81 performs zoom control or focuscontrol of the operative field camera 11. In the zoom control of theoperative field camera 11, either optical zoom control or digital zoomcontrol may be performed.

The CCU 81 includes an image processing unit 91.

The image processing unit 91 performs various kinds of image processingfor image display on the operative field image captured by the operativefield camera 11 and provides the display 14 with the operative fieldimage. Moreover, the image processing unit 91 includes a characteristicpoint detection unit 101.

The characteristic point detection unit 101 detects a predeterminedcharacteristic point from the operative field image captured by theoperative field camera 11 and provides the input control device 52 withthe information showing the characteristic point. For example, thecharacteristic point detection unit 101 detects a characteristic area inthe operative field image, such as a portion of an operation instrumentused in an operation, or an organ or blood vessel which is a biologicaltissue in an operative site. Note that the characteristic pointdetection unit 101 may detect the characteristic point by referring to,for example, a computed tomography (CT) image or external informationsuch as information from a navigation system for operation.

On the basis of the parameter from the parameter generation unit 62 andthe operation mode information from the operation mode determinationunit 63, the arm control unit 82 performs drive control of the cameraarm 12.

Note that, in the example in FIG. 5, the input control device 52 may beconfigured integrally with either the user I/F 51 or the control device71.

With the configuration above, in the operation system 10, the screenoperation is performed with the user's line-of-sight position or thecharacteristic point detected in an image of an operative site, regardedas the target point.

In a conventional operation system, a screen operation by aline-of-sight position allows a user to select an arbitrary position ina screen as a target point without using a hand. However, the user hasnot been able to operate the screen while comprehensively observing anoperative field, because the user is inevitably not able to observepositions other than a gaze point.

On the other hand, a screen operation regarding, as the target point,the characteristic point of an operation instrument or the like allowthe user to observe a position other than the target point. However,although an operation instrument needs to be moved for observation of awider operative field, the user has sometimes not been able to perform adesirable screen operation in a case where the movement of the operationinstrument is limited, for example, a case where an operation instrumentis holding an organ.

Therefore, it is desirable that the target point used for the screenoperation can be switched depending on a situation.

However, in some cases, there is a possibility that the switching of thetarget point by an input operation using a button or the like becomes aburden on the users. Especially, in a case where there is a plurality ofcandidates for the target point, switching of the target point requiresto be achieved by a simpler input operation in order to avoid users'confusion.

Therefore, operation control processing of the operation system 10,which achieves switching of the target point to be used for the screenoperation by a simple input operation without a burden on the users,will be described hereinafter.

<3. First Example of Operation Control Processing of Operation System>

First, a first example of operation control processing of the operationsystem 10 will be described with reference to a flowchart in FIG. 6. Inthis example, processing in a case where the MOVE operation is performedas the operation mode will be described.

In Step S11, the target point determination unit 61 determines whetheror not the line of sight of the operator 22 has been detected from theoperator's image from the user I/F 51. The processing of Step S11 isrepeated until the line of sight of the operator 22 is detected. Then,when it is determined that the line of sight of the operator 22 has beendetected, the processing proceeds to Step S12.

In Step S12, the characteristic point detection unit 101 detects thecharacteristic point from the operative field image captured by theoperative field camera 11.

In Step S13, the target point determination unit 61 determines whetheror not the characteristic point detected in the operative field image bythe characteristic point detection unit 101 and the line-of-sightposition of the operator 22 have matched with each other. When it isdetermined in Step S13 that the characteristic point and theline-of-sight position of the operator 22 have matched with each other,the processing proceeds to Step S14.

In Step S14, the target point determination unit 61 selects, as a targetpoint candidate, the characteristic point that has matched with theline-of-sight position of the operator 22.

Meanwhile, the processing proceeds to Step S15 in a case where it isdetermined in Step S13 that the characteristic point and theline-of-sight position of the operator 22 have not matched with eachother.

In Step S15, the target point determination unit 61 selects theline-of-sight position as the target point candidate.

In Step S16, the operation mode determination unit 63 determines whetheror not the foot switch 18 has been operated. Here, the operation of thefoot switch 18 is the operation to instruct an operation mode. For aperiod until the foot switch 18 is operated (for a period until anoperation mode is instructed), the processing in Steps S13 to S15 isrepeated, and selection of the target point candidate is repeated. Then,when it is determined that the foot switch 18 has been operated, theprocessing proceeds to Step S17.

In Step S17, trigged by the operation of the foot switch 18, the targetpoint determination unit 61 determines, as the target point, thecharacteristic point or the line-of-sight position, whichever isselected as the target point candidate.

In Step S18, the parameter generation unit 62 generates the parameter todisplay the operative field image in the display 14 with reference tothe target point determined by the target point determination unit 61.

In Step S19, the CCU 81 and the arm control unit 82 perform zoom controlof the operative field camera 11 and drive control of the camera arm 12,on the basis of the parameter generated by the parameter generation unit62 and the operation mode information showing the MOVE operation, theoperation mode information being provided by the operation modedetermination unit 63.

For example, the CCU 81 controls the drive of the camera arm 12 bycalculating an amount of the drive of the camera arm 12, the amountnecessary to move the target point to the center of the screen, on thebasis of information of an angle of view or the like of the operativefield camera 11.

In Step S20, the operation mode determination unit 63 determines whetheror not the foot switch 18 is being operated. While the foot switch 18 isbeing operated, the processing in Steps S16 and S17 is repeated. In thisexample, the MOVE operation is performed continuously with reference tothe target point.

Meanwhile, in a case where is determined in Step S20 that the footswitch 18 is not being operated, the zoom control of the operative fieldcamera 11 and the drive control of the camera arm 12 are stopped, andthe processing is ended.

For example, in the operative field image illustrated in the upper partof FIG. 7, a tip of an operation instrument 151 is detected as acharacteristic point P11, and a tip of an operation instrument 152 isdetected as a characteristic point P12. Here, in a case where theoperator 22 wishes to select the characteristic point P11 as the targetpoint from the characteristic point P11 and the characteristic pointP12, a line-of-sight position E11 of the operator 22 and thecharacteristic point P11 match with each other by the operator 22 gazingat the characteristic point P11.

Then, by the operator 22 in this state operating the foot switch 18 toinstruct the MOVE operation, as illustrated in the lower part of FIG. 7,the characteristic point P11 (the tip of the operation instrument 151)is positioned at the center of the operative field image.

Furthermore, in the operative field image illustrated in the upper partof FIG. 8, a portion of a blood vessel is detected as a characteristicpoint P21, and another portion of the blood vessel is detected as acharacteristic point P22. Here, in a case where the operator 22 wishesto select the characteristic point P21 as the target point from thecharacteristic point P21 and the characteristic point P22, aline-of-sight position E21 of the operator 22 and the characteristicpoint P21 match with each other by the operator 22 gazing at thecharacteristic point P21.

Then, by the operator 22 in this state operating the foot switch 18 toinstruct the MOVE operation, as illustrated in the lower part of FIG. 8,the characteristic point P21 is positioned at the center of theoperative field image.

Note that, in the example in FIG. 8, a CT image or external informationsuch as information from a navigation system for operation may be usedto detect the characteristic point of the blood vessel, organ, and thelike.

Furthermore, in the examples in FIGS. 7 and 8, the target point isdetermined by the foot switch 18 operated in a state that one of thecharacteristic points is selected as the target point candidate by theline-of-sight position. However, the line-of-sight position itself canbe determined as the target point in the processing described above.

For example, even in a case where there are selectable characteristicpoints as illustrated in the examples in FIGS. 7 and 8, theline-of-sight position itself is selected as the target point candidateby operating the foot switch 18 with the characteristic points not beingselected by the line-of-sight position. Furthermore, the line-of-sightposition itself may be determined as the target point by operating thefoot switch 18 with the characteristic points not being detected in theoperative field image.

By using the processing described above, triggered by the user'soperation, any one of the user's line-of-sight position with respect tothe operative field image or the characteristic point detected in theoperative field image is determined as the target point. Thus, switchingof the target point to be used for the screen operation can be achievedby a simple input operation without a burden on the user.

Note that, in the processing described above, information showing apoint to be the target point candidate such as the user's line-of-sightposition or the characteristic point of the operation instrument may beeither always or not always displayed in the operative field image.Especially, there is a possibility that presentation of informationirrelevant to an operative procedure in the operative field image usedfor an operation hinders progress of the operation. Therefore, in somecases, such information is preferable not to always be displayed.

FIG. 9 is a view illustrating examples of a feedback of the point to bethe target point candidate to the user.

In each operative field image in A and B of FIG. 9, the tip of theoperation instrument 151 is detected as the characteristic point P11,and the tip of the operation instrument 152 is detected as thecharacteristic point P12. In a case where any one of the characteristicpoints has matched with the line-of-sight position E11 of the user, thematching is fed back to the user.

In the example in A of FIG. 9, an area of the characteristic point P11is displayed with emphasis by the line-of-sight position E11 matchingwith the characteristic point P11. Furthermore, in the example in B ofFIG. 9, a voice message saying, for example, “You can select theoperation instrument 1.” is output by the line-of-sight position E11matching with the characteristic point P11. With these methods, thepoint to be the target point candidate can be recognized by the userwithout hindering progress of the operation.

Note that a method of feedback to the user is not limited to the methodsexemplified in FIG. 9.

Furthermore, in the processing described above, processing to forciblystop the operation control processing may be performed from a view pointof a fail-safe operation.

For example, there may be a case where, while the tip of the operationinstrument is determined as the target point, and the camera arm 12performs the following motion on the basis of the target point, the tipof the operation instrument is suddenly framed out of the operativefield image.

Therefore, as illustrated in the upper part of FIG. 10, after the tip ofthe operation instrument 151 (characteristic point P11) is determined asthe target point, coordinates of the characteristic point P11 on thescreen are always monitored.

Then, as illustrated in the lower part of FIG. 10, it is determinedwhether or not the coordinates of the characteristic point P11 arepositioned within a rectangular area 171 that is set on the screen andhas a size predetermined with reference to the center of the screen. Ina case where the coordinates of the characteristic point P11 aredetermined not to be positioned within the rectangular area 171, theoperation control processing is forced to stop.

Moreover, the operation control processing may be forced to stop also ina case where the user's line-of-sight position selecting the targetpoint candidate becomes unable to be detected during the followingmotion by the camera arm 12.

<4. Second Example of Operation Control Processing of Operation System>

Next, a second example of the operation control processing of theoperation system 10 will be described with reference to a flowchart inFIG. 11.

Note that description of the processing in Steps S31, S32, and S36 toS38 in the flowchart in FIG. 11 will be omitted, because the processingis similar to the processing in Steps S11, S12, and S18 to S20 in theflowchart in FIG. 6.

Furthermore, in the processing in FIG. 11, it is assumed that any one ofthe characteristic point in the operative field image or the user'sline-of-sight position is previously set as the target point.

In Step S33, the target point determination unit 61 determines whetheror not the characteristic point detected in the operative field image bythe characteristic point detection unit 101 and the line-of-sightposition of the operator 22 has matched with each other. The processingof Step S33 is repeated until the characteristic point and theline-of-sight position of the operator 22 match with each other. Then,when it is determined in Step S33 that the characteristic point and theline-of-sight position of the operator 22 have matched with each other,the processing proceeds to Step S34.

In Step S34, according to a selection by the user (operator 22), thetarget point determination unit 61 determines, as the target point, anyone of the characteristic point matching with the line-of-sight positionof the operator 22 or the line-of-sight position of the operator 22.

For example, in the operative field image illustrated in the upper partof FIG. 12, the tip of the operation instrument 151 is detected as thecharacteristic point P11, and the tip of the operation instrument 152 isdetected as the characteristic point P12. Here, it is assumed that theline-of-sight position E11 of the operator 22 is previously set as thetarget point. If the operator 22 in this state gazes at thecharacteristic point P11, the line-of-sight position E11 of the operator22 and the characteristic point P11 match with each other.

At this time, a pop-up window 181 appears in the operative field image.The pop-up window 181 prompts the operator 22 to select which one, theline-of-sight position E11 or the characteristic point P11, isdetermined as the target point. The pop-up window 181 includes a buttonto determine the line-of-sight position E11 as the target point and abutton to determine the characteristic point P11 (the tip of theoperation instrument 151) as the target point. A selected state of thesebuttons is toggled by, for example, operation of the foot switch 18, andeither of the buttons is finally selected.

In the example in FIG. 12, it is selected that the characteristic pointP11 is determined as the target point (the target point is switched fromthe line-of-sight position E11 to the characteristic point P11), and thecharacteristic point P11 (the tip of the operation instrument 151) ispositioned at the center of the operative field image, as illustrated inthe lower part of FIG. 12.

Note that, in the example in FIG. 12, the pop-up window 181 appears toprompt the operator 22 to select which one, the line-of-sight positionE11 or the characteristic point P11, is determined as the target point.However, not limited to this, the operator 22 may be notified of theprompt by, for example, sound.

Furthermore, by operation of the foot switch 18, which one, theline-of-sight position E11 or the characteristic point P11, isdetermined as the target point is selected. However, not limited tothis, which one, the line-of-sight position E11 or the characteristicpoint P11, is determined as the target point may be selected by usinganother method.

Thus, after the target point is determined, in Step S35, the targetpoint determination unit 61 determines whether or not the foot switch 18has been operated to instruct an operation mode. The processing of StepS35 is repeated until the foot switch 18 is operated. Then, when it isdetermined that the foot switch 18 has been operated, the processingproceeds to Step S36.

In and after Step S36, zoom control of the operative field camera 11 anddrive control of the camera arm 12 are performed according to theinstructed operation mode.

By using the processing described above, when the characteristic pointdetected in the operative field image and the user's line-of-sightposition with respect to the operative field image match with eachother, either the characteristic point or the user's line-of-sightposition, whichever is selected by the user's operation, is determinedas the target point. Therefore, switching of the target point to be usedfor the screen operation can be achieved by a simple input operationwithout a burden on the user.

Note that, in the processing described above, a display or notificationto prompt the operator 22 to select which one, the line-of-sightposition or the characteristic point, is determined as the target point,in other words, query about switching of the target point, is performedwhen the characteristic point and the line-of-sight position of theoperator 22 match with each other. However, the query about switching ofthe target point may be performed at any timing, such as at a timingspecified by the operator 22 or at a predetermined time interval.

Furthermore, the query about switching of the target point may beperformed in the operation control processing described with referenceto the flowchart in FIG. 6. In this case, for example, when threepoints, which are the user's line-of-sight position, the operationinstrument, and the biological tissue such as a blood vessel, arematched with one another, the query about switching the target point isperformed.

In the above, a configuration to move the target point to be the displayreference in the operative field image to the center of the screen isdescribed. However, not limited to this, the target point may be movedto any position on the screen.

Note that, in the operation system described above, an object to becontrolled on the basis of the input information is not limited to theoperative field camera 11 and the camera arm 12, and any otherconfiguration may be used.

In the above, an example is described in which the technology accordingto the present disclosure is applied to the operation system that usesthe operative field camera. However, the system to which the technologyaccording to the present disclosure may be applied is not limited tothis example. For example, the technology according to the presentdisclosure may be applied to an endoscope system or a microscopicoperation system.

Note that a hardware configuration for the control device 71 or theinput control device 52 can be configured with an information processingdevice including a circuit that can achieve a function of the controldevice 71 or the input control device 52. The information processingdevice constituting the control device 71 or the input control device 52includes, for example, a central processor unit (CPU), a read onlymemory (ROM), a random access memory (RAM), and a storage device. Thefunction of the control device 71 or the input control device 52 isachieved by the CPU developing and running, on the RAM, a programpreviously recorded in the ROM or the storage device, for example.

Moreover, the information processing device may include an external bussuch as a host bus, a bridge, and a peripheral componentinterconnect/interface (PCI) bus, an interface, an input device, anoutput device, a drive, a connection port, and a communication device.These components may be interconnected. Each of the components describedabove may include a general-purpose member or hardware dedicated to afunction of each component. Therefore, a configuration of hardware to beused can be appropriately changed according to a technical level at atime of implementing the present embodiment.

Note that a computer program to achieve each function of the informationprocessing devices, which constitute the operation system according tothe present embodiment as described above, can be manufactured andmounted on a personal computer, or the like. Furthermore, a recordingmedium, which stores such a computer program and is readable by acomputer, can also be provided. The recording medium is, for example, amagnetic disk, an optical disc, a magneto-optical disk, or a flashmemory. Furthermore, the computer program described above may bedistributed via, for example, a network, without using the recordingmedium. Furthermore, the number of computers to run the computer programis not especially limited. For example, a plurality of computers (forexample, a plurality of servers) may be linked one another to run thecomputer program.

Furthermore, the embodiment of the present technology is not limited tothe embodiment described above and can be changed in various wayswithout departing from the gist of the present technology.

Moreover, the present technology can have the following configurations.

(1)

An input control device including

a target point determination unit that determines a target point to be adisplay reference in an operative field image in a display, on the basisof a user's line-of-sight position with respect to the operative fieldimage displayed in the display and of a characteristic point detected inthe operative field image.

(2)

The input control device according to (1),

in which the target point determination unit determines any one of theuser's line-of-sight position with respect to the operative field imageor the characteristic point detected in the operative field image, asthe target point.

(3)

The input control device according to (1) or (2),

in which the characteristic point is a point that indicates a portion ofan operation instrument, the portion being detected in the operativefield image.

(4)

The input control device according to (1) or (2),

in which the characteristic point is a point that indicates a biologicaltissue in an operative site, the biological tissue being detected in theoperative field image.

(5)

The input control device according to any one of (2) to (4),

in which, in a case where the characteristic point detected in theoperative field image and the user's line-of-sight position with respectto the operative field image match with each other, the target pointdetermination unit determines, triggered by an operation by the user,the characteristic point as the target point.

(6)

The input control device according to (5),

in which, in a case where the characteristic point detected in theoperative field image and the user's line-of-sight position with respectto the operative field image do not match with each other, the targetpoint determination unit determines, triggered by the operation by theuser, the user's line-of-sight position as the target point.

(7)

The input control device according to any one of (2) to (4),

in which, in a case where the characteristic point detected in theoperative field image and the user's line-of-sight position with respectto the operative field image match with each other, the target pointdetermination unit determines either the characteristic point or theuser's line-of-sight position, whichever is selected by the user'soperation, as the target point.

(8)

The input control device according to any one of (1) to (7), furtherincluding

a parameter generation unit that generates a parameter to control amechanism that is driven to display the operative field image in thedisplay with reference to the determined target point.

(9)

The input control device according to (8), in which

the parameter is a parameter to perform at least one of zoom control ofa camera that captures the operative field image or drive control of acamera arm that supports the camera.

(10)

An input control method including

a step of determining, by an input control device, a target point to bea display reference in an operative field image in a display, on thebasis of a user's line-of-sight position with respect to the operativefield image displayed in the display and of a characteristic pointdetected in the operative field image.

(11)

An operation system including:

a camera that captures an operative field image;

a display that displays the operative field image; and

an input control device that includes

a target point determination unit that determines a target point to be adisplay reference in the operative field image in the display, on thebasis of a user's line-of-sight position with respect to the operativefield image displayed in the display and of a characteristic pointdetected in the operative field image.

REFERENCE SIGNS LIST

-   10 Operation system-   11 Operative field camera-   12 Camera arm-   13 Motion recognition camera-   14 Display-   15 Control device-   17 Microphone-   18 Foot switch-   51 User I/F-   52 Input control device-   61 Target point determination unit-   62 Parameter generation unit-   63 Operation mode determination unit-   71 Control device-   81 CCU-   82 Arm control unit-   91 Image processing unit-   101 Characteristic point detection unit

1. An input control device comprising a target point determination unitthat determines a target point to be a display reference in an operativefield image in a display, on a basis of a user's line-of-sight positionwith respect to the operative field image displayed in the display andof a characteristic point detected in the operative field image.
 2. Theinput control device according to claim 1, wherein the target pointdetermination unit determines any one of the user's line-of-sightposition with respect to the operative field image or the characteristicpoint detected in the operative field image, as the target point.
 3. Theinput control device according to claim 2, wherein the characteristicpoint is a point that indicates a portion of an operation instrument,the portion being detected in the operative field image.
 4. The inputcontrol device according to claim 2, wherein the characteristic point isa point that indicates a biological tissue in an operative site, thebiological tissue being detected in the operative field image.
 5. Theinput control device according to claim 2, wherein, in a case where thecharacteristic point detected in the operative field image and theuser's line-of-sight position with respect to the operative field imagematch with each other, the target point determination unit determines,triggered by an operation by the user, the characteristic point as thetarget point.
 6. The input control device according to claim 5, wherein,in a case where the characteristic point detected in the operative fieldimage and the user's line-of-sight position with respect to theoperative field image do not match with each other, the target pointdetermination unit determines, triggered by the operation by the user,the user's line-of-sight position as the target point.
 7. The inputcontrol device according to claim 2, wherein, in a case where thecharacteristic point detected in the operative field image and theuser's line-of-sight position with respect to the operative field imagematch with each other, the target point determination unit determineseither the characteristic point or the user's line-of-sight position,whichever is selected by the user's operation, as the target point. 8.The input control device according to claim 1, further comprising aparameter generation unit that generates a parameter to control amechanism that is driven to display the operative field image in thedisplay with reference to the determined target point.
 9. The inputcontrol device according to claim 8, wherein the parameter is aparameter to perform at least one of zoom control of a camera thatcaptures the operative field image or drive control of a camera arm thatsupports the camera.
 10. An input control method comprising a step ofdetermining, by an input control device, a target point to be a displayreference in an operative field image in a display, on a basis of auser's line-of-sight position with respect to the operative field imagedisplayed in the display and of a characteristic point detected in theoperative field image.
 11. An operation system comprising: a camera thatcaptures an operative field image; a display that displays the operativefield image; and an input control device that includes a target pointdetermination unit that determines a target point to be a displayreference in the operative field image in the display, on a basis of auser's line-of-sight position with respect to the operative field imagedisplayed in the display and of a characteristic point detected in theoperative field image.